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ABSTRACT:  The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  Force  (JTF)  is  chartered  by  Office  of  the 
Secretary  of  Defense  to  investigate  the  utility  of  Advanced  Distributed  Simulation  (ADS)  Technology  to  Test  and 
Evaluation  (T&E).  JADS  is  executing  three  test  programs  (  C4I,  Precision  Guided  Munitions,  and  Electronic  Warfare) 
representing  slices  of  the  overall  T&E  spectrum  as  well  as  observing  other  activity  within  the  T&E  community  to  form  its 
conclusions.  One  of  the  slices,  the  Electronic  Warfare  test,  is  using  HLA.  To  understand  expected  latency  prior  to  the  test 
execution,  JADS  is  building  a  testbench  to  integrate  and  test  the  hardware  and  principle  software  components.  This  paper 
discusses  the  JADS  plan  to  test  the  wide  area  network  system  latency  prior  to  building  the  network  and  hosting  the  self¬ 
protection  jammer  (SPJ)  test  federation.  The  primary  focus  of  this  paper  is  on  the  process  and  tools  used  by  JADS  to 
characterize  the  RTI/network  interactions,  the  system  characterization  measurements  made  in  the  testbed,  and  how  that 
characterization  will  be  fed  back  into  the  federation  design. 


1.  Introduction: 

HLA  is  a  rapidly  evolving  product.  As  it  continues  to  mature,  there  are  a  number  of  questions  that  users  must  address  as 
they  build  their  federations,  particularly  if  the  federation  requires  high  performance  from  the  communications 
infrastructure.  Some  of  these  questions  are:  “How  do  I  determine  what  performance  I  need  from  the  RTI  for  my 
federation?”,  “How  do  I  communicate  that  to  potential  RTI  suppliers?”,  and  “How  do  I  determine  that  the  RTI  I’ve 
received  meets  my  requirements?”  These  questions  will  become  more  relevant  in  the  future  as  the  RTI  evolves  into  a 
more  commercial-like  shrink-wrapped  product. 

The  federation  and  the  communications  infrastructure  are  a  complex  system  designed  to  meet  the  sponsor’s  objective. 
Complex  systems  require  good  system  engineering  process  or  a  lot  of  luck  to  succeed.  The  Defense  Systems  Management 
College  uses  the  diagram  shown  in  Figure  1.0  to  depict  the  systems  engineering  process.  The  requirements  analysis  and 
functional  analysis/allocation  parts  of  the  system  engineering  process  are  covered  in  the  first  three  steps  of  the  Five-Step 
FEDEP  process.  Synthesis  occurs  in  step  4  of  the  FEDEP  process,  federation  integration  and  test,  supported  by  associated 
W&A  and  applicable  test  activities.  All  this  suggests  that  federation  development  effort  based  upon  performance  driven 
requirements  may  be  expected  to  bounce  back  and  forth  across  the  first  four  steps  of  the  FEDEP  as  the  federation  design  is 
iterated  through  the  system  engineering  process.  It  should  be  obvious  from  this  discussion  that  we  feel  RTI  testing  alone, 
while  interesting  and  informative,  needs  to  be  part  of  an  overall  plan.  RTI  testing  for  high  performance  federations  must 
be  accomplished  in  concert  with  a  full  characterization  of  the  other  communications  infrastructure  components,  the 
federates,  and  the  Federation  Object  Model.  Furthermore,  if  the  results  of  the  characterizations  indicate  performance 
shortfalls,  those  results  need  to  be  fed  back  into  the  sponsor  needs  and  federation  objectives  as  well  as  the  federation 
design. 


Figure  1.0  System  Engineering  Process 

A  sensible  systems  engineering  approach  also  provides  a  basis  for  determining  the  performance  a  federation  needs  from 
the  communications  infrastructure  and  the  RTI.  The  JADS  derived  requirements  are  presented  as  part  of  this  paper  to 
provide  context  for  the  test  discussion,  but  the  primary  focus  of  this  paper  is  RTI  testing.  The  rest  of  this  paper  will  focus 
on  the  suite  of  tests  that  will  be  used  to  determine  that  the  Wide  Area  Network  (WAN)  equipment  and  RTI  1.3  meet  the 
Joint  Advanced  Distributed  Simulation  (JADS)  Electronic  Warfare  (EW)  test  federation  requirements.  This  discussion 
begins  by  presenting  a  quick  introduction  of  the  JADS  test  program,  the  objectives  of  the  EW  test  federation,  and  a  brief 
description  of  the  derived  RTI  requirements.  With  that  background,  the  RTI  test  philosophy  and  objectives  are  discussed, 
followed  by  a  discussion  of  the  RTI  test  approach. 

2.  Background 

JADS  is  an  OSD  sponsored  Joint  Test  Force  chartered  to  determine  the  utility  of  Advanced  Distributed  Simulation  (ADS) 
technology  for  Test  and  Evaluation  (T&E)  of  military  systems.  JADS  is  doing  this  by  looking  at  three  slices  of  the  T&E 
spectrum.  One  of  those  slices  is  the  JADS  EW  Self  Protection  Jammer  test.  This  test  will  use  two  HLA  compliant 
federations  to  recreate  an  actual  Open  Air  Range  (OAR)  test  of  an  EW  system  under  test  (an  AN/ALQ-13 1  Block  II  self¬ 
protection  jammer)  to  see  how  the  test  results  using  the  HLA  federations  correlate  with  the  OAR  results. 

The  OAR  test  represents  an  effectiveness  test  of  the  airborne  self  protection  jammer.  One  referent  test  condition  will  be 
repeated  to  gather  a  large  sample  of  jammer  effectiveness  data.  The  environment,  the  threat  systems,  and  the  jammer  are 
all  instrumented  to  allow  standard  measures  of  performance  to  be  calculated  and  to  allow  the  engagements  to  be  recreated 
in  the  federations.  During  the  test  phases  using  the  HLA  federations,  each  OAR  pass  will  be  recreated,  the  federate 
interactions  will  be  monitored,  and  the  measures  of  performance  will  be  calculated  in  real-time. 

2.1  JADS  Performance  Requirements 

Closed  loop  testing  using  distributed  simulation  technology  runs  the  risk  that  the  communications  infrastructure  used  to 
carryout  the  interactions  between  objects  will  change  the  outcome  either  through  lost  interactions,  or  by  changing  the 
temporal  nature  of  the  interaction.  This  temporal  change  is  usually  an  increase  in  the  time  for  the  interaction  to  occur  and 
is  called  latency.  The  amount  of  allowable  latency  depends  on  the  nature  of  the  interactions  and  the  decision  cycle  of  each 
system.  The  EW  test  interaction  of  interest  is  the  threat  radar  activation,  jammer  identification  and  response,  and 
associated  threat  response.  Depending  on  how  the  engagement  is  carried  out,  the  interaction  can  be  the  jammer’s 
computer  working  against  the  threat  computer  or  the  jammer’s  computer  working  against  the  threat’s  human  operator. 
Due  to  limitations  in  the  threat  radar  federate,  we  will  be  looking  at  the  latter  interaction  exclusively.  Therefore,  the 
limitations  that  we  have  placed  on  the  communication  infrastructure  latency  is  500  milliseconds.  That  means  that  from 
the  time  that  the  radar  changes  state,  the  infrastructure  has  250  milliseconds  to  get  that  message  to  the  jammer  and 
another  250  milliseconds  to  return  the  jammer’s  response.  The  jammer’s  decision  cycle  time  is  excluded  from  the  500 
millisecond  latency.  We  refer  to  this  as  an  “end-to-end  interaction”  during  the  EW  test  event.  Of  the  250  milliseconds, 
the  RTI  is  allocated  150  milliseconds.  It  is  important  to  note  that  this  requirement  is  valid  only  for  the  JADS  EW  test. 
The  CROSSBOW  sponsored  Threat  Simulator  Linking  Activity  (TSLA)  Network  Specification  indicates  that  the  JADS 
latency  budget  is  not  as  restrictive  as  other  types  of  testing  will  require.  Each  type  of  HLA  federation  will  have  to  make 
their  own  assessment  of  critical  performance  factors  and  determine  key  metrics  for  acceptable  federation  performance. 


3.  RTI  Test  Philosophy 

The  RTI  test  philosophy  simply  stated  is  to  operate  the  RTI  in  a  series  of  hardware/software  configurations  to  determine 
RTI  performance  as  variables  which  may  impact  performance  are  changed.  Each  RTI  test  environment  is  designed  to 
build  upon  previous  testing.  Together  the  environments  form  a  series  of  filter  screens  that  increase  user  confidence  in  RTI 
performance  while  isolating  increasingly  complex  federate  interactions  for  examination. 

The  first  environment  is  a  simple  computer-to-computer  architecture  intended  to  mimic  tests  done  by  RTI  developers. 
The  environments  increase  in  complexity  until  the  entire  federation  network  architecture  (except  for  T-l  lines)  is  in  place. 
These  represent  development  type  tests  that  are  designed  to  determine  RTI  baseline  performance.  The  final  test  is  an 
operational  type  test  that  will  use  the  federates,  (or  acceptable  surrogates),  message  structures,  and  data  that  we  will  be 
using  in  the  actual  federate  execution.  At  this  time  we  will  also  be  characterizing  the  network  bandwidth  utilization  and 
latency.  Once  we  are  convinced  the  network  will  meet  our  needs,  we  will  bring  the  T-l  connections  up  and  repeat  a 
portion  of  the  testing  using  the  long  haul  communications  network. 

During  the  entire  RTI  test,  we  will  be  treating  the  RTI  as  a  black  box  component  that  has  been  added  into  a  known 
network  architecture.  We’ve  chosen  that  approach  to  best  imitate  a  future  customer  that  has  little  or  no  insight  into  the 
RTI  providers  software  development  and  testing  process. 

We  have  not  identified  all  the  software  tools  and  test  stubs  that  will  be  used  during  each  phase  of  the  testing.  Our  intent  is 
to  reuse  DMSO  test  tools  as  well  as  test  stubs  of  our  own  creation  to  allow  comparison  of  results  between  like 
hardware/software  configurations  and  validation  of  the  developed  test  stubs. 

4.  RTI  Test  Objectives 

The  primary  objective  of  the  RTI  testing  is  to  ensure  that  JADS  has  an  acceptable  communications  infrastructure  for  each 
test  federation  in  order  to  recreate  the  OAR  test  environment.  Acceptable  means  that  all  hardware  and  software 
components  are  behaving  as  required  and  that  the  total  system  latency  is  within  budget  over  the  expected  range  of 
message  rates  and  sizes  used  to  recreate  the  OAR  test  event  interactions.  RTI  testing  is  necessary  for  the  JADS  EW  test 
program  because  the  federations  are  producing  EW  performance  data  that  will  be  carefully  measured  and  compared 
between  the  test  phases.  Any  variability  in  test  results  introduced  by  RTI,  the  other  communications  infrastructure 
components,  the  federates,  or  the  Federation  Object  Model  must  be  identified  and  measured  prior  to  actual  EW  test  events. 

In  order  to  feed  RTI  test  results  back  into  the  federation  design  we  must  understand  the  behavior  of  the  RTI  around  our 
expected  performance  envelope.  The  results  of  the  RTI  testing  will  be  used  to  guide  any  simulation  object  model  changes 
that  may  be  required  to  meet  the  total  end-to-end  interaction  latency  requirement.  It  may  be  possible  to  make  minor 
modifications  to  the  message  structures,  update  rates,  interactions,  or  other  aspects  of  the  infrastructure  to  optimize 
federation  communication.  Such  tuning  should  be  planned  for  in  the  development  of  high  performance  federations. 

5.  RTI  Test  Approach 

As  stated  earlier,  the  RTI  will  be  operated  in  a  series  of  controlled  test  environments.  The  first  environment  uses  an  SGI 
02  5000  and  an  SGI  02  10000  running  IRIX  6.3  connected  via  a  dedicated  Ethernet  connection.  The  first  test  set  will 
use  DMSO  test  tools  to  establish  baseline  performance  and  functionality  of  the  system.  The  second  test  set  will  use  the 
test  stubs  and  procedures  that  will  be  used  for  the  rest  of  the  RTI  testing.  These  stubs  are  simple  variations  of  the  data 
playback  federates.  The  set  begins  with  fixed  message  update  rate,  a  fixed  tick  rate,  and  variable  message  size.  The 
message  size  is  then  fixed  and  the  message  rate  is  allowed  to  vary.  Then  both  the  message  rate  and  size  are  fixed  and  the 
message  is  passed  first  as  an  attribute  update  and  then  as  an  interaction.  Finally,  the  tick  rate  is  varied.  Throughout  all 
these  tests,  the  total  RTI  induced  latency  (time  from  entrance  into  the  RTI  until  time  of  exit)  is  being  measured.  While 
this  paper  concentrates  on  describing  a  methodology  for  JADS  RTI  testing,  the  reader  should  be  aware  of  the  fact  that 
associated  communications  link  throughput,  latency,  and  hardware/software  configurations,  as  examples,  are  also  being 
tested.  All  sources  of  possible  latency  are  being  measured  through  a  disciplined  process  of  adjusting  one  variable  at  a  time 
and  collecting  elapsed  time  data  for  the  same  transaction  in  differing  reference  test  conditions.. 

The  second  environment  uses  the  same  computers  connected  as  shown  in  Figure  5.1  to  form  two  network  segments.  The 
test  set  will  follow  the  same  process  as  the  previous  test  set  using  the  JADS  test  stubs.  This  hardware  will  be 
implemented,  baseline  performance  will  be  measured,  and  RTI  functionality  will  be  verified  prior  to  the  start  of 


performance  testing.  The  test  progression  will  be  the  same  as  the  first  environment.  Message  rate,  size, 
attribute/interaction,  and  tick  are  each  examined  around  the  values  specified  in  the  JADS  EW  test  inputs  in  the  RTI 
Performance  Workbook. 


SGI  02 


Figure  5.1  Two  Segment  Network  Architecture 

The  third  environment  uses  three  computers  on  three  network  segments  to  match  the  network  topology  we  will  be  using  in 
our  test.  The  topology  is  depicted  below  in  Figure  5.2.  The  segment  to  segment  connections  use  the  components  shown 
in  Figure  5.1.  Again  RTI  functionality  is  established  prior  to  performance  testing.  This  time,  another  variable  is 
examined.  After  the  attribute/interaction  test,  fixed  rate  and  size  messages  are  passed  using  first  reliable  transmission, 
then  best  effort  to  determine  the  impact  of  transmission  type  on  system  latency. 


Figure  5.2  Three  Segment  Topology 


Finally,  three  additional  computers  are  added  to  one  of  the  segments  to  represent  the  distribution  of  the  multiple  federates 
across  the  network.  Network  performance  and  RTI  functionality  are  again  measured.  This  time  no  additional 
performance  testing  is  accomplished.  This  test  set  concludes  the  developmental  lype  testing. 

Operational  type  testing  is  accomplished  by  replacing  the  test  stubs  with  either  the  actual  federate  or  a  surrogate  for  the 
federate.  The  concept  is  to  allow  all  the  federates  to  interact  according  to  how  they  will  interact  during  the  test.  The 
testing  will  begin  with  two  federates  interacting  and  build  to  all  six  interacting  using  actual  data  (where  practical)  in  the 
real  scenario.  Finally,  the  network  will  be  stress  tested  by  adding  objects,  most  likely  missiles,  into  the  scenario  until 
system  performance  becomes  unacceptable  due  to  failure  or  excessive  latency.  It  should  be  noted  that  for  the  purposes  of 
the  JADS  EW  test,  the  objects  don’t  have  to  behave  correctly,  they  just  need  to  exist  and  pass  updates  to  the  subscribing 
federates. 

After  the  successful  completion  of  the  development  and  operational  testing,  the  testbed  will  be  dismantled,  shipped  and 
reassembled  as  the  wide  area  network  to  be  used  during  the  actual  test.  Subsets  of  the  development  and  operational 
testing  will  be  re-accomplished  to  verify  the  operation  of  the  network  and  to  characterize  its  performance.  The  final 
criteria  to  be  met  will  be  a  graduation  exercise  by  the  network  and  JADS  EW  federates  that  will  use  actual  sample  test 
data  and  will  cover  all  the  observed  interactions  from  the  Open  Air  Range  phase  of  the  JADS  EW  test. 

6.0  Conclusion 

High  performance  federations  must  understand  the  communications  infrastructure  upon  which  they  are  built.  An  integral 
part  of  the  infrastructure  is  the  RTI.  RTI  testing  for  a  federation  must  be  accomplished  in  concert  with  network  testing 
and  should  be  a  systematic  characterization  designed  to  identify  the  inherent  performance  of  the  network.  That 
characterization  can  then  be  used  to  tune  the  federation  object  model  so  that  the  performance  desired  can  be  achieved. 
JADS  EW  is  developing  a  test  plan,  procedures,  and  test  stubs  to  accomplish  communications  infrastructure 


characterization.  We  will  execute  the  characterization  using  RTI  1.3  and  use  the  results  to  tune  the  FOM. 
Characterization  will  also  be  used  in  the  W&A  of  the  federation  prior  to  execution.  Results  and  lessons  learned  will  be 
provided  to  DMSO  and  the  HLA  user  community. 
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